Focus stacking image processing apparatus, imaging system, and image processing system

ABSTRACT

An image processing apparatus comprises: an image acquisition unit for acquiring a plurality of original images acquired by imaging a specimen including a structure in various focal positions using a microscope apparatus; an image generation unit for generating, on the basis of the plurality of original images, a first image on which blurring of an image of the structure has been reduced in comparison with the original images; and an analysis unit for obtaining information relating to the structure included in the first image by applying image analysis processing to the first image. The image generation unit selects a part of the original images having focal positions included within a smaller depth range than a thickness of the specimen from the plurality of original images obtained from the specimen, and generates the first image using the selected original images.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an image processing apparatus, an imagingsystem, and an image processing system, and in particular to a techniquefor assisting observation of an object with the use of a digital image.

2. Description of the Related Art

Recently, a virtual slide system attracts attention in the field ofpathology, as a successor to an optical microscope which is currentlyused as a tool for pathological diagnosis. The virtual slide systemenables pathological diagnosis to be performed on a display by imaging aspecimen (a sample) to be observed placed on a slide and digitizing theimage. The digitization of pathological diagnosis images with thevirtual slide system makes it possible to handle conventional opticalmicroscope images of specimens as digital data. It is expected this willbring about various merits, such as more rapid remote diagnosis,provision of information to patients through digital images, sharing ofdata of rare cases, and more efficient education and training.

When using a virtual slide system, it is required to digitize an entireimage of a specimen to be observed placed on a slide in order to realizeequivalent performance to that of an optical microscope. Thedigitization of the entire image of the specimen makes it possible toexamine the digital data generated with the virtual slide system byusing viewer software running or a PC or work station. The digitizedentire image of the specimen will generally constitute an enormousamount of data, from several hundred million pixels to several billionpixels when represented by the number of pixels.

Even though the amount of data generated by the virtual slide system isenormous, this makes it possible to examine the specimen image eithermicroscopically (in enlarged detail views) or macroscopically (inoverall perspective views) by scaling the image with the viewer, whichprovides various advantages and conveniences. All the necessaryinformation can be preliminarily acquired so that images of anyresolution and any magnification can be displayed instantaneously asrequested by a user. Further, by performing image analysis on theobtained digital data in order to comprehend a cell shape, calculate acell count, or calculate an area ratio (an N/C ratio) between cytoplasmand a nucleus, for example, various information useful for pathologicaldiagnosis can also be provided.

Incidentally, an imaging optical system of a virtual slide system isdesigned with an emphasis on resolving power and therefore has anextremely shallow depth of field. Accordingly, a range in which focus isachieved relative to the thickness of a specimen serving as an object tobe imaged is extremely narrow, and therefore images of tissues and cellspositioned away from a focal position in a depth direction (a directionalong an optical axis of the imaging optical system or a directionperpendicular to an observation surface of a slide) are blurred. It istherefore difficult to observe an entire specimen from a singletwo-dimensional image. Further, in an image including a large amount ofblur, the precision of characteristic amount extraction and imagerecognition decreases, leading to a reduction in the reliability ofimage analysis performed by a computer.

An image processing method known as focus stacking is available as amethod of solving this problem. Focus stacking is a method of generatingan image having a deep depth of field from a plurality of imagesobtained by imaging an object in various focal positions. JapanesePatent Application Publication No. 2005-037902, for example, discloses asystem in which a deep-focus image is generated by dividing imageshaving different focal positions respectively into a plurality ofsections and performing focus stacking in each section.

According to the method disclosed in Japanese Patent ApplicationPublication No. 2005-037902, an image that is in focus as a whole andincludes little blur can be obtained. However, although this type ofdeep-focus image is useful for rough observation of the specimen as awhole, it is not suitable for detailed observation of a part of thespecimen or comprehension of a three-dimensional structure and athree-dimensional distribution of tissues, cells, and so on. The reasonfor this is that when focus stacking is performed, depth directioninformation is lost, and therefore a user cannot determine front-rearrelationships between respective structures (cells, nuclei, and so on)in the image simply by viewing the image. Further, when structuresoriginally existing in different depth direction positions areoverlapped on the image at an identical contrast, it is difficult toseparate and identify the structures not merely through visualobservation but even through image analysis using a computer.

SUMMARY OF THE INVENTION

The present invention has been designed in view of these problems, andan object thereof is to provide a technique for preserving depthdirection information relating to a specimen so that the specimen can beobserved using a digital image, and generating an image suitable forimage analysis processing using a computer.

The present invention in its first aspect provides an image processingapparatus comprising: an image acquisition unit for acquiring aplurality of original images acquired by imaging a specimen including astructure in various focal positions using a microscope apparatus; animage generation unit for generating, on the basis of the plurality oforiginal images, a first image on which blurring of an image of thestructure has been reduced in comparison with the original images; andan analysis unit for obtaining information relating to the structureincluded in the first image by applying image analysis processing to thefirst image, wherein the image generation unit selects a part of theoriginal images having focal positions included within a smaller depthrange than a thickness of the specimen from the plurality of originalimages obtained from the specimen, and generates the first image usingthe selected original images.

The present invention in its second aspect provides an imaging systemcomprising: a microscope apparatus for obtaining a plurality of originalimages by imaging a specimen including a structure in various focalpositions; and the image processing apparatus according to the firstaspect, which obtains the plurality of original images from themicroscope apparatus.

The present invention in its third aspect provides an image processingsystem comprising: an image server for storing a plurality of originalimages obtained by imaging a specimen including a structure in variousfocal positions; and the image processing apparatus according to thefirst aspect, which obtains the plurality of original images from theimage server.

The present invention in its fourth aspect provides a computer programstored on a non-transitory computer readable medium, the program causinga computer to perform a method comprising the steps of: acquiring aplurality of original images acquired by imaging a specimen including astructure in various focal positions using a microscope apparatus;generating, on the basis of the plurality of original images, a firstimage on which blurring of an image of the structure has been reduced incomparison with the original images; and acquiring information relatingto the structure included in the first image by applying image analysisprocessing to the first image, wherein, in the image generation step, apart of the original images having focal positions included within asmaller depth range than a thickness of the specimen is selected fromthe plurality of original images obtained from the specimen, and thefirst image is generated using the selected original images.

According to this invention, depth direction information relating to aspecimen can be preserved so that the specimen can be observed using adigital image, and an image suitable for image analysis processing by acomputer can be generated.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overall view showing a layout of apparatuses in an imagingsystem according to a first embodiment;

FIG. 2 is a functional block diagram of an imaging apparatus accordingto the first embodiment;

FIG. 3 is a conceptual diagram illustrating focus stacking;

FIG. 4 is a conceptual diagram illustrating a relationship between aspecimen and depth and distance information;

FIG. 5 is a flowchart showing an overall flow of image processingaccording to the first embodiment;

FIG. 6 is a flowchart showing a flow of image generation processingaccording to the first embodiment;

FIG. 7 is a flowchart showing a flow of focus stacking processingaccording to the first embodiment;

FIG. 8 is a flowchart showing a flow of image analysis according to thefirst embodiment;

FIG. 9 is an overall view showing a layout of apparatuses in an imageprocessing system according to a second embodiment;

FIG. 10 is a flowchart showing an overall flow of image processingaccording to the second embodiment; and

FIG. 11 is a flowchart showing a flow of focus stacking processingaccording to a third embodiment.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

FIG. 1 is an overall view showing a layout of apparatuses in an imagingsystem according to a first embodiment of the invention.

The imaging system according to the first embodiment is composed of animaging apparatus (microscope apparatus) 101, an image processingapparatus 102, and a display device 103, and is a system with a functionto acquire and display a two-dimensional image of a specimen (a testsample) as an object to be imaged. The imaging apparatus 101 and theimage processing apparatus 102 are connected to each other with adedicated or general-purpose I/F cable 104. The image processingapparatus 102 and the display device 103 are connected to each otherwith a general-purpose I/F cable 105.

The imaging apparatus 101 is a virtual slide apparatus having a functionof acquiring a plurality of two-dimensional images at different focalpositions in an optical axis direction and outputting digital images.The acquisition of the two-dimensional images is done with a solid-stateimaging device such as a CCD (Charge Coupled Device) or CMOS(Complementary Metal Oxide Semiconductor). Alternatively, the imagingapparatus 101 may be formed by a digital microscope apparatus having adigital camera attached to an eye piece of a normal optical microscope,in place of the virtual slide apparatus.

The image processing apparatus 102 is an apparatus having a function forgenerating an analysis image suitable for image analysis from aplurality of original images obtained from the imaging apparatus 101, afunction for generating an observation image suitable for visualobservation, a function for applying image analysis processing to theanalysis image, and so on. The image processing apparatus 102 is formedby a general-purpose computer or work station having hardware resourcessuch as a CPU (central processing unit), a RAM, a storage device, anoperation unit, and an I/F. The storage device is a mass informationstorage device such as a hard disk drive, in which a program forexecuting processing steps to be described later, data, an OS (operatingsystem), and so on are stored. The above-mentioned functions arerealized by the CPU downloading a program and data required for the RAMfrom the storage device and executing the program. The operation unit isformed by a keyboard or a mouse, and is used by an operator to inputvarious types of instructions. The display device 103 is a monitor whichdisplays an image for observation and an image analytical resultobtained by the arithmetic processing done by the image processingapparatus 102, and is formed by a CRT, a liquid-crystal display, or thelike.

Although in the example show in FIG. 1, the imaging system consists ofthree components: the imaging apparatus 101, the image processingapparatus 102, and the display device 103, the invention is not limitedto this configuration. For example, the image processing apparatus maybe integrated with the display device, or the functions of the imageprocessing apparatus may be incorporated in the imaging apparatus.Further, the functions of the imaging apparatus, the image processingapparatus and the display device can be realized by a single apparatus.Conversely, the functions of the image processing apparatus and the likecan be divided so that they are realized by a plurality of apparatusesor devices.

(Configuration of Imaging Apparatus)

FIG. 2 is a block diagram illustrating a functional configuration of theimaging apparatus 101.

The imaging apparatus 101 is schematically composed of an illuminationunit 201, a stage 202, a stage control unit 205, an imaging opticalsystem 207, an imaging unit 210, a development processing unit 216, apre-measurement unit 217, a main control system 218, and an externalinterface 219.

The illumination unit 201 is means for irradiating a slide 206 placed onthe stage 202 with uniform light, and is composed of a light source, anillumination optical system, and a drive control system for the lightsource. The stage 202 is drive-controlled by the stage control unit 205,and is movable along three axes of X, Y, and Z. The optical axisdirection shall be defined as the Z direction. The slide 206 is a memberin which a tissue section or smeared cell to be examined is applied on aslide glass and encapsulated under a cover glass together with anencapsulant.

The stage control unit 205 is composed of a drive control system 203 anda stage drive mechanism 204. The drive control system 203 performs drivecontrol of the stage 202 in accordance with an instruction received fromthe main control system 218. A direction and amount of movement, and soon of the stage 202 are determined based on position information andthickness information (distance information) on the specimen obtained bymeasurement by the pre-measurement unit 217 and a instruction from theuser. The stage drive mechanism 204 drives the stage 202 according tothe instruction from the drive control system 203.

The image-formation optical system 207 is a lens group for forming anoptical image of the specimen in the preparation 206 on an imagingsensor 208.

The imaging unit 210 is composed of the imaging sensor 208 and an analogfront end (AFE) 209. The imaging sensor 208 is a one-dimensional ortwo-dimensional image sensor for converting a two-dimensional opticalimage into an electric physical amount by photoelectric conversion, anda CCD or CMOS, for example is used as the imaging sensor 208. When theimaging sensor 208 is a one-dimensional sensor, a two-dimensional imagecan be obtained by scanning the image in a scanning direction. Theimaging sensor 208 outputs an electrical signal having a voltage valueaccording to an intensity of light. When a color image is desired as acaptured image, a single-plate image sensor having a Bayer arrangementcolor filter attached thereto can be used.

The AFE 209 is a circuit for converting an analog signal output from theimaging sensor 208 into a digital signal. The AFE 209 is composed of anH/V driver, a CDS, an amplifier, an AD converter, and a timing generatoras described later. The H/V driver converts a vertical synchronizingsignal and horizontal synchronizing signal for driving the imagingsensor 208 into a potential required to drive the sensor. The CDS(correlated double sampling) is a correlated double sampling circuit forremoving noise from fixed pattern. The amplifier is an analog amplifierfor adjusting gain of the analog signal the noise of which has beenremoved by the CDS. The AD converter converts an analog signal into adigital signal. When the final stage output of the system has eightbits, the AD converter converts an analog signal into digital data whichis quantized to about 10 to 16 bits in consideration of processing to bedone in the subsequent stage, and outputs this digital data. Theconverted sensor output data is referred to as RAW data. The RAW data issubjected to development processing in the subsequent developmentprocessing unit 216. The timing generator generates a signal foradjusting timing of the imaging sensor 208 and timing of the subsequentdevelopment processing unit 216.

When a CCD is used as the imaging sensor 208, the AFE 209 describedabove is indispensable. However, when a CMOS image sensor capable ofdigital output is used as the imaging sensor 208, the sensor includesthe functions of the AFE 209. Although not shown in the drawing, animaging control unit for controlling the imaging sensor 208 is provided.This imaging control unit performs not only control of operation of theimaging sensor 208 but also control of operation timing such as shutterspeed, frame rate, and ROI (Region of Interest).

The development processing unit 216 is composed of a black correctionunit 211, a white balance adjustment unit 212, a demosaicing processingunit 213, a filter processing unit 214, and a γ correction unit 215. Theblack correction unit 211 performs processing to subtractblack-correction data obtained during light shielding from each pixel ofthe RAW data. The white balance adjustment unit 212 performs processingto reproduce desirable white color by adjusting the gain of each colorof RGB according to color temperature of light from the illuminationunit 201. Specifically, white balance correction data is added to theblack-corrected RAW data. This white balance adjustment processing isnot required when a monochrome image is handled.

The demosaicing processing unit 213 performs processing to generateimage data of each color of RGB from the RAW data of Bayer arrangement.The demosaicing processing unit 213 calculates a value of each color ofRGB for a pixel of interest by interpolating values of peripheral pixels(including pixels of the same color and pixels of other colors) in theRAW data. The demosaicing processing unit 213 also performs correctionprocessing (complement processing) for defective pixels. The demosaicingprocessing is not required when the imaging sensor 208 has no colorfilter and an image obtained is monochrome.

The filter processing unit 214 is a digital filter for performingsuppression of high-frequency components contained in an image, noiseremoval, and enhancement of feeling of resolution. The γ correction unit215 performs processing to add an inverse to an image in accordance withgradation representation capability of a commonly-used display device,or performs gradation conversion in accordance with human visualcapability by gradation compression of a high brightness portion or darkportion processing. Since an image is acquired for the purpose ofmorphological observation in the present embodiment, gradationconversion suitable for the subsequent image combine processing ordisplay processing is performed on the image.

Development processing functions in general include color spaceconversion for converting an RGB signal into a brightnesscolor-difference signal such as a YCC signal, and processing to compressmass image data. However, in this embodiment, the RGB data is useddirectly and no data compression is performed.

Although not shown in the drawings, a function of peripheral darkeningcorrection may be provided to correct reduction of amount of light inthe periphery within an imaging area due to effects of a lens groupforming the imaging optical system 207. Alternatively, variouscorrection processing functions for the optical system may be providedto correct various aberrations possibly occurring in the imaging opticalsystem 207, such as distortion correction for correcting positionalshift in image formation or magnification color aberration correction tocorrect difference in magnitude of the images for each color.

The pre-measurement unit 217 is a unit for performing pre-measurement aspreparation for calculation of position information of the specimen onthe slide 206, information on distance to a desired focal position, anda parameter for adjusting the amount of light attributable to thethickness of the specimen. Acquisition of information by thepre-measurement unit 217 before main measurement makes it possible toperform efficient imaging. Further, designation of positions in which tostart and terminate imaging (a focal position range) and an imaginginterval (an interval between focal positions; also referred to as a Zinterval) when obtaining images having different focal positions is alsoperformed on the basis of the information generated by thepre-measurement unit 217. A two-dimensional imaging sensor having alower resolving power than the imaging sensor 208 is used to obtainposition information relating to a two-dimensional plane. Thepre-measurement unit 217 learns the position of the specimen on an XYplane from the obtained images. A laser displacement meter or ameasurement instrument employing a Shack-Hartmann method is used toobtain distance information and thickness information. A method ofobtaining the specimen thickness information will be described below.

The main control system 218 has a function to perform control of theunits described so far. The functions of the main control system 218 andthe development processing unit 216 are realized by a control circuithaving a CPU, a ROM, and a RAM. Specifically, a program and data arestored in the ROM, and the CPU executes the program using the RAM as awork memory, whereby the functions of the main control system 218 andthe development processing unit 216 are realized. The ROM may be formedby a device such as an EEPROM or flush memory, and the RAM may be formedby a DRAM device such as a DDR3.

The external interface 219 is an interface for transmitting an RGB colorimage generated by the development processing unit 216 to the imageprocessing apparatus 102. The imaging apparatus 101 and the imageprocessing apparatus 102 are connected to each other through an opticalcommunication cable. Alternatively, an interface such as a USB orGigabit Ethernet (registered trademark) can be used.

A flow of imaging processing in the main measurement will be brieflydescribed. The stage control unit 205 positions the specimen on thestage 202 based on information obtained by the pre-measurement such thatthe specimen is positioned for imaging. Light emitted by theillumination unit 201 passes through the specimen and the imagingoptical system 207 thereby forms an image on the imaging surface of theimaging sensor 208. An output signal from the imaging sensor 208 isconverted into a digital image (RAW data) by the AFE 209, and this RAWdata is converted into a two-dimensional RGB image by the developmentprocessing unit 216. The two-dimensional image thus obtained istransmitted to the image processing apparatus 102.

The configuration and processing as described above enable acquisitionof a two-dimensional image of the specimen at a certain focal position.A plurality of two-dimensional images with different focal positions canbe obtained by repeating the imaging processing by means of the stagecontrol unit 205 while shifting the focal position in a direction of theoptical axis (Z direction). A group of images with different focalpositions obtained by the imaging processing in the main measurementshall be referred to as “Z-stack images”, and two-dimensional imagesforming the Z-stack images at the respective focal positions shall bereferred to as the “layer images” or “original images”.

Although the present embodiment has been described in terms of anexample in which a single-plate method is used to obtain a color imageby means of an image sensor, a three-plate method of obtaining a colorimage using three RGB image sensors can be used instead of thesingle-plate method. Alternatively, a triple imaging method can be usedin which a single image sensor and a three-color light source are usedtogether and imaging is performed three times while switching the colorof the light source.

(Regarding Focus Stacking)

FIG. 3 is a conceptual diagram of focus stacking. The focus stackingprocessing will be schematically described with reference to FIG. 3.Images 501 to 507 are seven-layer images which are obtained by imagingseven times an object including a plurality of structures atthree-dimensionally different spatial positions while sequentiallychanging the focal position in the optical axis direction (Z direction).Reference numerals 508 to 510 indicate structures contained in theacquired image 501. The structure 508 comes into focus at the focalposition of the image 503, but is out of focus at the focal position ofthe image 501. Therefore, it is difficult to comprehend theconfiguration of the structure 508 in the image 501. The structure 509comes into focus at the focal position of the image 502, but is slightlyout of focus at the focal position of the image 501. Therefore, it ispossible, though not satisfactory, to comprehend the configuration ofthe structure 509 in the image 501. The structure 510 comes into focusat the focal position of the image 501 and hence the structure thereofcan be comprehended sufficiently in the image 501.

In FIG. 3, the structures which are blacked out indicate those in focus,the structures which are white indicate those slightly out of focus, andthe structures represented by the dashed lines indicate those out offocus. Specifically, the structures 510, 511, 512, 513, 514, 515, and516 are in focus in the images 501, 502, 503, 504, 505, 506, and 507,respectively. The description of the example shown in FIG. 3 will bemade on the assumption that the structures 510 to 516 are located atdifferent positions in the horizontal direction.

An image 517 is an image obtained by cutting out respective regions ofthe structures 510 to 516 which are in focus in the images 501 to 507and merging these regions. By merging the focused regions of theplurality of images as described above, a focus-stacked image which isfocused in the entirety of the image can be obtained. This processingfor generating an image having a deep depth of field by the digitalimage processing is referred to also as focus stacking. Further, amethod of selecting and merging regions that are in focus and have ahigh contrast, as shown in FIG. 3, is referred to as a select and mergemethod. In this embodiment, an example in which focus stacking isperformed using this select and merge method will be described. However,another focus stacking method may of course be used.

(Re: Thickness of Specimen)

FIG. 4 is a conceptual diagram illustrating a relationship between thespecimen and the depth and distance information. The concept of a depthrange, which is a feature of this embodiment, will now be describedusing this drawing.

FIG. 4 shows a cross-section of a slide in pattern form. The slide is astructure in which a specimen 603 is fixed between a transparent slideglass 602 and a transparent cover glass 601. A sealant constituted by atransparent adhesive exists between the cover glass 601 and the specimen603.

604 denotes a distance from a reference position 605 to an upper surfaceof the cover glass 601. Further, 606 denotes a distance from a referenceposition 607 to a lower surface of the slide glass 602. These distances604 and 606 may be measured using a laser displacement meter or thelike, for example.

608 denotes a thickness of the specimen 603. It is difficult to measurethe specimen thickness 608 directly, and therefore the thickness 608 ispreferably calculated by subtracting the distances 604 and 606, a coverglass thickness 609, and a slide glass thickness 610 from an intervalbetween the reference positions 605 and 607. Note that undulation occurson the cover glass 601 and the slide glass 602 due to a gripping method,the influence of the sealant, and variation in the thickness of thespecimen. Therefore, to measure the specimen thickness 608 with a highdegree of precision, distance information is preferably obtained in aplurality of positions in a two-dimensional plane (an XY plane),whereupon an average or an intermediate value of the distanceinformation is taken.

The cover glass thickness 609 can be measured or set at a prescribedvalue registered in advance. When the cover glass thickness 609 ismeasured, measurement is preferably performed at a plurality of points,taking into account the undulation of the cover glass 601. When aprescribed value is used, on the other hand, it may be assumed that novariation occurs in the thickness of the cover glass 601.

The slide glass thickness 610, similarly to the cover glass thickness609, can be measured on the basis of the measurement result or set at aprescribed value registered in advance. The slide glass 602 is typicallylarger than the cover glass 601, and therefore the slide glass thickness610 may be calculated by measuring a distance from the referenceposition 605 to an upper surface of the slide glass 602 and subtractinga total value of the measurement result and the distance 606 from theinterval between the reference positions 605 and 607.

611 a to 611 d are structures included in the specimen 603. It isassumed here that 611 a to 611 d respectively indicate cell nuclei. Thenucleus 611 b is severed on a perpendicular plane to the optical axis.The nuclei 611 b and 611 c have different optical axis direction (Zdirection) positions (depths) but partially overlap when seen from theoptical axis direction (i.e. when projected onto an XY plane).

612 denotes a lower surface of the cover glass 601, or in other words asurface of the specimen 603. By aligning the focal position of theimaging optical system with this position 612, an image of the surfaceof the specimen 603 can be captured. Note that in reality, undulation ofthe slide, variation in the thickness of the specimen 603, and so onmean that even when the focal position is aligned with the position 612,the entire surface region of the specimen 603 is not in focus, andin-focus regions intermix with blurred regions. This applies likewisewhen the focus is aligned with any position on the optical axis.Further, Z direction positions in which structures (cell nuclei, and soon) to be subjected to observation and analysis exist are distributedrandomly, as shown in FIG. 4, and it may therefore be impossible toobtain sufficient information for a pathological diagnosis simply byimaging the specimen in a single focal position.

In a conventional system, therefore, a deep-focus image is typicallygenerated by obtaining a plurality of layer images while graduallyshifting the focal position within a depth range between the cover glasslower surface and the slide glass upper surface, for example, andperforming focus stacking on all of the obtained layer images. However,although image blur can be greatly reduced on a combined image obtainedusing this method, depth direction information relating to the specimenis completely lost, making it impossible to grasp front-rearrelationships (vertical relationships) between the nuclei 611 a to 611d. Further, an image of the nucleus 611 b overlaps (merges with) animage of the nucleus 611 c on the combined image, making it difficult todistinguish between the two nuclei 611 b and 611 c. When this type ofimage overlap occurs, an undesirable reduction in precision may occurduring image analysis processing such as comprehending cell shapes,calculating a cell count, and calculating an area ratio (an N/C ratio)between the cytoplasm and the nucleus, for example.

Hence, in the system according to this embodiment, a combined imagesuitable for image analysis is provided by performing focus stacking onan “analysis image” used for image analysis using only a part of thelayer images (in a smaller depth range than the thickness of thespecimen) rather than all of the layer images. The analysis image is animage on which blur has been reduced appropriately in comparison withthe layer images (original images). Further, this system is capable ofgenerating an “observation image” (a second image) suitable for visualobservation by a user separately from the analysis image (a firstimage). The observation image is an image subjected to a smaller degreeof blur reduction than the analysis image. In other words, by preservingan appropriate amount of blur in images of structures removed from thefocal position on the observation image, the vertical relationshipsbetween the structures can be grasped easily in an intuitive manner.Note that when the extent of the blur (in other words, the depth offield) on a layer image is suitable for observation, the layer imageserving as an original image may be used as is as the observation image.

Hence, a feature of this system is that an image having an appropriatedepth of field (or contrast) is generated automatically in accordancewith the application (image analysis by a computer or visual observationby a user), enabling an improvement in user-friendliness.

Here, the analysis image and the observation image differ from eachother in the size of the depth range in which the layer images to beused for focus stacking are selected. Specifically, the depth range ofthe analysis image is larger than the depth range of the observationimage. In other words, the number of layer images used for focusstacking is larger in the analysis image than in the observation image.Alternatively, it may be said that the analysis image has a deeper depthof field than the observation image and that the degree of blurreduction is greater (the amount of blur is smaller) in the analysisimage than in the observation image. The depth range of the analysisimage can be determined appropriately in accordance with the size of thestructure to be analyzed or the type of diagnosis (cytological diagnosisor tissue diagnosis). More specifically, the depth range is preferablydetermined to be approximately equal to the size of the structure (cell,nucleus, or the like) to be identified (detected) during image analysis.In the case of a specimen used for a tissue diagnosis, for example, thesize of a nucleus of a normal cell is approximately 3 to 5 μm and thesize of a nucleus of an abnormal cell is up to approximately 10 μm, andtherefore the depth range is preferably set at not less than 3 μm andnot more than 10 μm. When the focal position interval (the Z interval)between layer images is 1 μm and no undulation exists on the cover glass601, approximately 3 to 10 layer images are used in focus stacking ofthe analysis image.

(System Operations)

FIG. 5 is a flowchart showing an overall flow of image processingperformed by the imaging system. A flow of processing for capturing thelayer images and processing for generating the observation image and theanalysis image will now be described using this drawing. It is assumedthat the pre-measurement processing is terminated before the mainmeasurement processing, and that a low resolution, XY plane image of theslide is transmitted together with Z direction distance and thicknessinformation to the main control system 218 before the processing shownin FIG. 5 begins.

In Step S701, the main control system 218 detects a range in which thespecimen exists from the XY plane image obtained in the pre-measurementusing well-known image processing such as edge detection or objectrecognition. The range detected here is designated as an imaging rangeof the main measurement. By limiting (reducing) the imaging range on thebasis of the result of the pre-measurement in this manner, rather thanimaging the entire slide, reductions in a processing time and a dataamount can be achieved.

In Step S702, the main control system 218 designates a Z directionimaging range on the basis of the Z direction distance information andthickness information of the slide, obtained in the pre-measurement.More specifically, an imaging start position (the cover glass lowersurface, for example), an imaging end position (the slide glass uppersurface, for example), and the imaging interval (the Z interval) arepreferably designated. The imaging interval can be determined on thebasis of the depth of field of the imaging optical system 207. Forexample, when an image is considered to be in focus within a focalposition range of ±0.5 μm (i.e. when the depth of field is 1 μm), theimaging interval is preferably set to be equal to or smaller than 1 μm.The imaging interval may be fixed or varied over the imaging range. Forexample, the specimen thickness differs between a cytological diagnosisand a tissue diagnosis (several tens of μm in the case of a cytologicaldiagnosis and several μm in the case of a tissue diagnosis), andtherefore the imaging interval is preferably set to be wider in the caseof a cytological diagnosis than in the case of a tissue diagnosis. Bywidening the imaging interval, the number of imaging operationsdecreases, leading to a reduction in the number of obtained layerimages, and as a result, the imaging time can be shortened and the dataamount can be reduced.

Information relating to the imaging range obtained in Steps S701 andS702 is transmitted respectively to the stage control unit 205, theimaging unit 210, and the development processing unit 216.

Next, a two-dimensional image (a layer image) is captured in each focalposition.

First, in Step S703, the stage control unit 205 moves the stage 202 inthe X and Y directions to achieve positioning between the imaging rangeof the specimen and an angle of view of the imaging optical system 207and the imaging sensor 208. Further, the stage control unit 205 movesthe stage 202 in the Z direction to align the focal position on thespecimen with the imaging start position.

In Step S704, the illumination unit 201 illuminates the specimen and theimaging unit 210 takes in an image. In this embodiment, a sensor havinga Bayer array is envisaged, making it possible to advance to asubsequent operation following a single imaging operation. This applieslikewise to a three plate system. In the case of a triple imaging systemin which a light source is switched, the light source is switchedbetween R, G, and B in an identical position, and after obtaining therespective images, the routine advances to the next step.

In Step S705, imaging data are processed by the development processingunit 216 to generate an RGB image, whereupon the generated RGB image istransmitted to the image processing apparatus 102. The image isprimarily stored in an internal storage of the imaging apparatus 101,whereupon the routine may advance to a transmission step. Through theprocessing of S703 to S705, a single layer image captured in a singlefocal position can be obtained.

In Step S706, a determination is made as to whether or not imaging inall focal positions is complete (in other words, whether or not thefocal position has reached the imaging end position). When imaging iscomplete, or in other words when acquisition of all of the layer imagesis complete, the routine advances to Step S707. When the focal positionhas not yet reached the imaging end position, the routine returns toS703, where the focal position is shifted by the imaging intervaldesignated in S702 and the next imaging operation is performed. When,for example, the distance between the imaging start position and theimaging end position is 30 μm and the imaging interval is 1 μm, theprocessing of S703 to S706 is performed 31 times so that 31 layer imagesare obtained.

In Step S707, the image processing apparatus 102, having received theresults of acquisition of all of the layer images, performs varioussetting operations relating to image generation processing. Here, thenumber of obtained layer images is learned, and information such asdistance information required to generate the observation image and theanalysis image and a depth of field range is obtained and set.

In Step S708, the image processing apparatus 102 generates theobservation image and the analysis image on the basis of values set inStep S707. This operation will be described in detail below using FIG.6.

Step S709 onward corresponds to an example of processing using theobservation image and the analysis image. In Step S709, the imageprocessing apparatus 102 determines whether to perform processing on theobservation image or the analysis image. The routine advances to StepS710 when processing is to be performed on the observation image and toStep S711 when processing is to be performed on the analysis image. Notethat in this flowchart, processing of the observation image andprocessing of the analysis image are executed exclusively, but the twotypes of processing may be executed in parallel or in sequence.

In Step S710, the image processing apparatus 102 obtains the observationimage and displays the obtained observation image on the display device103. The observation image may be an unprocessed layer image selectedeither by the user or automatically from the plurality of layer images,or a combined image subjected to focus stacking in the depth directioninformation acquisition range of S708.

In Step S711, the image processing apparatus 102 obtains the analysisimage to be subjected to processing. In Step S712, the image processingapparatus 102 implements image analysis processing on the basis of theselected analysis image. The image analysis processing will be describedin detail below using FIG. 8.

In Step S713, the image processing apparatus 102 displays the result ofthe image analysis processing (S712) on the display device 103. At thistime, the analysis result is preferably presented as supplementaryinformation to the observation image displayed in Step S710. Theanalysis result may be displayed alongside the observation image,overlaid onto the observation image, or displayed in another form.

Note that the flowchart of FIG. 5 illustrates the flow of a series ofoperations including capture of the layer images and generation,analysis, and display of the observation image and the analysis image.However, the operations of the system are not limited to this example,and for example, the imaging processing of S701 to S706, the imagegeneration processing of S707 to S708, and the analysis/displayprocessing of S709 to S713 may be performed separately at differenttimes. For example, the imaging processing may be executed alone inbatches on a plurality of slides housed in a stacker (not shown),whereupon Z-stack images (layer image groups) of the respective slidesare stored in an internal storage device of the image processingapparatus 102 or a storage device on a network. As regards the imagegeneration processing and the analysis/display processing, the user canselect a desired image from the storage device and perform processingthereon at a desired time.

(Image Generation Processing)

FIG. 6 is a flowchart showing a flow of the image generation processing.Here, Step S708 of FIG. 5 will be described in detail.

In Step S801, the image processing apparatus 102 determines whether animage generation subject is the observation image or the analysis image.The routine advances to Step S802 in the case of the observation imageand to Step S807 in the case of the analysis image. Note that theflowchart illustrates an example in which observation image generationprocessing and analysis image generation processing are executedexclusively, but in reality, both the observation image and the analysisimage are generated, and therefore the two types of generationprocessing are executed in parallel or in sequence.

(1) Observation Image Generation Processing

In Step S802, the user selects one image from the plurality of layerimages corresponding to the plurality of focal positions. For example,the user is asked to designate a focal position or presented with apreview screen on which the plurality of images are arranged and askedto select an image therefrom.

In Step S803, a determination is made as to whether or not the imageselected in Step S802 has a sufficient depth of field for visualobservation. When the depth of field is sufficient, the routine jumps toStep S806 (in this case, the layer image is used as is as theobservation image). When the depth of field is not sufficient, or inother words when the depth information is acknowledged to be incomplete,the routine advances to Step S804. Note that the determination as towhether or not the depth of field is sufficient may be made by the userby viewing a preview of the image.

In Step S804, the user designates the range (depth range) of the layerimages used to generate the observation image. At this time, the usermay designate either a depth range or a number of images. In apreferable configuration, a plurality of preview images formed bycombining two images, three images, and so on may be displayed on thescreen, for example, and the user may be asked to select a desirednumber of combined images. Note that the preview images may be formedfrom focus-stacked rough images or images created using simplercombining processing (addition, a blending, or the like) than focusstacking.

In Step S805, the image processing apparatus 102 selects the pluralityof layer images within the designated depth range and implements focusstacking processing thereon. The focus stacking processing will bedescribed in detail below using FIG. 7.

In Step S806, the image processing apparatus 102 designates the imageselected in Step S802 or the focus-stacked image generated in Step S805as the observation image. The observation image is then stored in theinternal storage device of the image processing apparatus 102 or apredetermined storage device on a network.

(2) Analysis Image Generation Processing

In Step S807, the image processing apparatus 102, having learned thatthe image generation subject is an analysis application, selects animage (to be referred to as a reference image) that is to serve as areference position during focus stacking from the plurality of layerimages. Reference image selection may be performed by the user. Here,the reference position can be set arbitrarily, but the lower surface ofthe cover glass, or in other words an upper side (an imaging opticalsystem side) surface of the specimen is preferably selected as thereference position. The reason for this is that in a normal image, bothstructures existing above the focal position (focal surface) andstructures existing below the focal position are superimposed onto theimage as blurred images, whereas in an image formed by aligning thefocal position with the specimen surface, only transparent objects suchas the sealant and the cover glass exist above the focal position, andtherefore a blur component is halved (i.e. only the lower sidestructures are blurred). A clear image exhibiting little blur is moresuitable for image analysis. Alternatively, when the structure to beanalyzed is known, an image on which the structure is most in focus maybe selected as the reference image, and when the depth at which analysisis to be performed (for example, the center of the specimen, X μm fromthe specimen surface, or the like) has already been determined, an imagehaving that depth may be selected.

In Step S808, the range (depth range) of the layer images used togenerate the analysis image is designated. The depth range is set suchthat the reference position designated in S807 forms an upper end, acenter, or a lower end of the depth range. The depth range may bedesignated by the user, but is preferably determined automatically bythe image processing apparatus 102 in accordance with the size of theanalysis subject, the aim of the analysis, and so on. For example, whencalculating the N/C ratio, which is the area ratio between the cytoplasmand the nucleus, during a tissue diagnosis, the depth range ispreferably set at not less than 3 μm and not more than 10 μm, takinginto account that a diameter of the nucleus is between approximately 3and 5 μm in a normal cell and expands to approximately several timesthat as a result of nuclear enlargement and multinucleation. Further, inthe case of a cytological diagnosis, the aim is usually to obtain anoverall picture of an exfoliated cell having a size (thickness) ofapproximately several tens of μm in order to grasp the cell shape, andtherefore the depth range is preferably set at approximately 20 μm. Itis assumed that correspondence relationships between the size of thedesired depth range and the analysis subject and analysis aim are set inadvance in the image processing apparatus 102.

Further, in Step S808, the Z direction interval (Z interval) between theimages to be used during focus stacking may be set in addition to thedepth range. For example, the depth range used during a cytologicaldiagnosis is larger than that of a tissue diagnosis, and therefore, whenall of the images in the depth range are used for focus stacking, theprocessing time increases. Hence, by increasing the Z interval in caseswhere the depth range is large, the size of the analysis subject islarge, and so on, the number of images is reduced, leading to areduction in processing time.

In Step S809, the image processing apparatus 102 selects the pluralityof layer images included in the depth range set in S807 and S808. InStep S810, the image processing apparatus 102 implements focus stackingprocessing using the selected layer images. The focus stackingprocessing is identical in content to that of Step S805, and will bedescribed below using FIG. 7.

In Step S811, the image processing apparatus 102 designates thefocus-stacked image generated in Step S810 as the analysis image. Theanalysis image is then stored in the internal storage device of theimage processing apparatus 102 or a predetermined storage device on anetwork.

As a result of the processing described above, both the observationimage and the analysis image can be obtained from identical Z-stackimages (a layer image group).

(Focus Stacking Processing)

FIG. 7 is a flowchart showing a flow of the focus stacking processing.Here, Steps S805 and S810 of FIG. 6 will be described in detail.

In Step S901, the image processing apparatus 102 obtains the pluralityof layer images selected as focus stacking subjects. As described above,a larger number of images is used during focus stacking of the analysisimage than during focus stacking of the observation image.

In Step S902, the image processing apparatus 102 divides each of theobtained images into a plurality of small regions of a predeterminedsize. The size of the divided region is determined taking into accountthe size of the structure (cell, nucleus, or the like) to be subjectedto observation or analysis. For example, the size of the divided regionis preferably set such that a length of one side of the divided regionis between approximately half the diameter and the entire diameter ofthe observation or analysis subject structure.

In Step S903, the image processing apparatus 102 detects a contrastvalue in relation to each divided region of each image. A method ofdetermining frequency components by performing a discrete cosinetransform in each divided region, determining a sum of high frequencycomponents within the frequency components, and using this sum as avalue expressing a degree of contrast may be cited as an example ofcontrast detection. More simply, a difference between a maximum valueand a minimum value of brightness values in the divided region may bedetermined as the contrast value, or a value obtained by calculating anedge amount using an edge detection spatial filter may be set as thecontrast value. Various known methods may be applied to contrastdetection.

In Step S904, the image processing apparatus 102 creates a contrast map.The contrast map is a table having an identical number of elements tothe number of divided regions. A contrast value and an image number of acorresponding divided region are mapped to each element as map values.For example, in a case where the image is divided into 100×100 regionsin S902, the contrast map has 100×100 elements. In Step S904, anarbitrary image (the image on the upper end of the depth range or thelike, for example) is selected from the plurality of images obtained inS901, and the contrast value and image number of the image are inputinto the contrast map as initial values.

In Step S905, the image processing apparatus 102 selects a differentimage to the image selected in Step S904 as a comparison subject image.

In Step S906, the image processing apparatus 102 compares the contrastvalues of the comparison subject image and the contrast map. When thecontrast value of the comparison subject image is larger, the routineadvances to Step S907. When the contrast value of the comparison subjectimage is smaller or when the contrast values of the two images areidentical, the routine skips the processing of Step S907 and advances toStep S908.

In Step S907, the image processing apparatus 102 writes the contrastvalue and image number of the comparison subject image to the contrastmap (updates the contrast map). The contrast value comparison of S906and updating of the contrast map in S907 are performed for each dividedregion.

In Step S908, the image processing apparatus 102 determines whether ornot the comparison processing has been implemented on all of the imagesselected in Step S901. When the comparison processing has been performedon all of the images, the routine advances to Step S909. When theprocessing is not complete, the routine returns to Step S905, where thecomparison processing is repeated. As a result, the number of the imagehaving the highest contrast value is recorded for each divided region onthe completed contrast map.

In Step S909, the image processing apparatus 102 extracts a dividedimage from the layer image having the corresponding image number in eachdivided region by referring to the contrast map.

In Step S910, the image processing apparatus 102 implements stitchingprocessing to merge the divided images extracted in Step S909. Byperforming the steps described above, a combined image merging highcontrast regions, or in other words sharp, focused regions, can begenerated from the plurality of layer images.

(Image Analysis Processing)

FIG. 8 is a flowchart showing a flow of image analysis. Here, Step S712of FIG. 5 will be described in detail.

In Step S1101, the image processing apparatus 102 obtains the analysisimage subjected to focus stacking for the purpose of analysis. Here, atissue diagnosis will be described as an example, and accordingly, athinly sliced, HE (hematoxylin and eosin) stained tissue fragment isused as the specimen.

In Step S1102, the image processing apparatus 102 extracts edges of acell serving as the analysis subject included in the analysis image.Here, processing is performed to extract regions having a red to pinkcolor gamut, using the fact that the cell is stained red to pink by theeosin. In the analysis image according to this embodiment, image blur isreduced by the focus stacking, and therefore edge extraction andsubsequent contour extraction can be performed with a high degree ofprecision. Note that in order to perform the extraction processing withan even higher degree of precision, edge emphasis processing using aspatial filter may be implemented on the analysis image in advance. Theedge extraction described here is, in actuality, cell membranedetection.

In Step S1103, the image processing apparatus 102 extracts cell contourson the basis of the edges extracted in Step S1102. When the edgesextracted in Step S1102 are discontinuous and intermittent, a continuouscontour can be extracted by implementing processing to connect theedges. A typical linear interpolation method may be used to connect thediscontinuous edges, but to achieve greater precision, a high orderinterpolation method may be used.

In Step S1104, the image processing apparatus 102 recognizes andspecifies individual cells on the basis of the contours detected in StepS1103. A cell is typically circular and the size thereof is more or lessfixed. Therefore, erroneous determinations of cells can be reduced byusing knowledge information such as shape and size. Further, in thisembodiment, the depth range used to generate the analysis image is setat an appropriate range on the basis of the size of the nucleus of thecell, and therefore overlap of cells existing at different depths on theimage is minimized. The processing to recognize and specify the cellscan therefore be performed with a high degree of precision. Note thatthe possibility of partial cell overlap remains, and therefore cellspecification may be difficult. In this case, the recognition andspecification processing may be implemented again after receiving aresult of subsequent cell nucleus specification processing.

In Step S1105, the image processing apparatus 102 extracts the contoursof the cell nuclei. In HE staining, the nucleus of the cell is stainedviolet by the hematoxylin and the peripheral cytoplasm is stained red bythe eosin. Hence, in Step S1105, processing is performed to detect apart having a violet central part and a red periphery and extract aboundary between the violet region and the red region.

In Step S1106, the image processing apparatus 102 specifies the cellnucleus on the basis of the contour information detected in Step S1105.In a normal cell, the nucleus typically has a size of approximately 3 to5 μm, but when an abnormality occurs, various changes such as sizeenlargement, multinucleation, and deformation occur. One indication ofthe existence of the nucleus is that it exists in the cell specified inStep S1104. Cells that could not easily be specified in Step S1104 canbe determined by specifying the nucleus.

In Step S1107, the image processing apparatus 102 measures the sizes ofthe cells and the cell nuclei specified in Step S1104 and Step S1106.Here, the size means the surface area, and therefore the surface area ofthe cytoplasm in the cell membrane and the interior surface area of thenucleus are respectively determined. Further, the total number of cellsmay be counted, and statistical information relating to the shapes andsizes thereof may be obtained.

In Step S1108, the image processing apparatus 102 calculates the N/Cratio, which is the area ratio between the cytoplasm and the nucleus, onthe basis of the surface area information obtained in Step S1107.Statistical information relating to the calculation results of theindividual cells is then obtained.

In Step S1109, the image processing apparatus 102 determines whether ornot analysis processing has been performed on all of the cells withinthe range of the analysis image, or in certain cases a range designatedby the user. When the analysis processing is complete, the overallprocessing is complete. When the analysis processing is not complete,the routine returns to Step S1102, from where the analysis processing isrepeated.

By performing the steps described above, image analysis useful fordiagnostic support can be implemented.

Advantages of this Embodiment

According to this embodiment, as described above, two images, namely theobservation image and the analysis image, can be generated on the basisof a plurality of layer images having different focal positions. Theobservation image has a shallower depth of field than the analysisimage, and therefore structures (cell, nuclei, and the like) removedfrom the depth of field form blurred images on the image. When the userviews the image visually, the blurred images serve as depth informationenabling the user to grasp the three-dimensional structure andthree-dimensional distribution of the object. With regard to theanalysis image, on the other hand, overlap of the analysis subjects onthe image can be minimized by performing focus stacking in anappropriate range corresponding to the size of the analysis subject andthe aim of the analysis, and therefore image analysis processing can beperformed easily and precisely. Hence, with the system according to thisembodiment, two images can be generated from identical Z-stack images (alayer image group) in accordance with the application and aim, andtherefore an improvement in user-friendliness can be achieved.

Further, the focus stacking is performed using a plurality of layerimages, and therefore artifacts are less likely to appear in comparisonwith an image obtained simply by applying depth recovery processing suchas edge emphasis on a single-depth image having a shallow depth offield. As a result, a high-quality image that can be used to provide aprecise diagnosis can be generated. Moreover, the Z-stack images can beobtained by a simple process of moving a stage or an imaging devicecarrying the specimen in the optical axis direction, and an imagingapparatus including such a mechanism can be realized comparativelyeasily.

Furthermore, in this embodiment, focus stacking is performed using aselect and merge method, and this method involves simpler processingthan other methods, such as a spatial frequency filtering method to bedescribed in a third embodiment. As a result, a circuit scale and acalculation amount can also be suppressed.

Note that in this embodiment, the image generation and image analysisprocessing is performed by the image processing apparatus 102 after theZ-stack images are captured by the imaging apparatus 101, but theprocessing procedures are not limited thereto, and by linking theimaging apparatus 101 and the image processing apparatus 102, imagingcan be performed at a required timing. For example, when it isdetermined that the layer images required by the image processingapparatus 102 to generate an image suitable for the aim are insufficientor nonexistent, the imaging apparatus 101 can be notified of therequired imaging range (XY range, Z position, and Z interval). In sodoing, image acquisition can be performed in a shorter imaging time andwith a smaller data amount.

Second Embodiment

An image processing system according to a second embodiment of thepresent invention will now be described using the drawings.

In the first embodiment, an example in which the layer images used togenerate the observation image and the analysis image are obtained asrequired by the imaging apparatus was described. In the secondembodiment, an example in which the layer images are obtained in advanceand the image processing apparatus obtains required layer images from animage server during image generation will be described. The secondembodiment also differs from the first embodiment in that pre-processingrelating to image stacking is varied between a cytological diagnosis anda tissue diagnosis. The following description focuses on thesedifferences.

FIG. 9 is an overall view showing a layout of apparatuses in the imageprocessing system according to the second embodiment.

The image processing system according to this embodiment includes animage server 1201, the image processing apparatus 102, and the displaydevice 103. The image processing apparatus 102 is capable of obtaining atwo-dimensional image (a layer image) of a specimen from the imageserver 1201 and displaying the obtained image. The image server 1201 andthe image processing apparatus 102 are connected to each other by ageneral-purpose I/F LAN cable 1203 via a network 1202. The image server1201 is a computer having a large-capacity storage device that storesZ-stack images captured by an imaging apparatus (a virtual slideapparatus). The image server 1201 stores, in addition to image data,data relating to pre-measurement performed by the imaging apparatus. Theimage processing apparatus 102 and the display device 103 are similar tothose of the first embodiment.

In the example shown in FIG. 9, the image processing system isconstituted by three apparatuses, namely the image server 1201, theimage processing apparatus 102, and the display device 103, but thepresent invention is not limited to this layout. For example, an imageprocessing apparatus having an integrated display device may be used, orthe functions of the image processing apparatus may be incorporated intothe image server. Further, the functions of the image server, the imageprocessing apparatus, and the display device may be realized by a singleapparatus. Conversely, the functions of the image server and the imageprocessing apparatus may be divided among a plurality of apparatuses.

(System Operations)

FIG. 10 is a flowchart showing an overall flow of image processingperformed by the image processing apparatus. The flow of processing forgenerating the observation image and the analysis image, which is afeature of this embodiment, will now be described using this drawing.Note that similar processes to the first embodiment have been allocatedidentical reference symbols, and description thereof has been omitted.

In Step S1301, the image processing apparatus 102 reads an arbitraryimage (for example, an image having an uppermost focal position) fromthe layer image group relating to the subject specimen to be used forimage generation from the image server 1201, and displays a preview ofthe image on the display device 103. The user is then asked to designatean XY range (a range required to create the observation image and theanalysis image) in which the observation and analysis subject structureexists on the preview image.

In Step S1302, the image processing apparatus 102 determines an imagingrange in the Z direction (optical axis direction) of the layer image tobe used for image generation on the basis of AF information relating tothe layer image group. The AF information is information relating to thefocal position of each image, which is created during imaging using anautofocus function (for example, a function for detecting a focal pointusing a frequency component or a contrast value of an image) of theimaging apparatus. In this embodiment, the AF information is stored inthe image server 1201 together with the layer image group.

In Step S1303, the image processing apparatus 102 determines whether thesubject specimen is to be used in a cytological diagnosis or a tissuediagnosis. The user may be asked to designate a cytological diagnosis ora tissue diagnosis, but the image processing apparatus 102 is capable ofdetermining a cytological diagnosis or a tissue diagnosis automaticallyfrom the thickness and the staining method of the subject specimen. Thethickness of the specimen is typically between approximately 4 and 5 μmin a tissue diagnosis and at least several tens of μm in a cytologicaldiagnosis. As regards staining, meanwhile, HE staining is typically usedin a tissue diagnosis, whereas Papanicolaou staining is typically usedin a cytological diagnosis, and it is therefore possible to infer fromthe tinge of the specimen whether a tissue diagnosis or a cytologicaldiagnosis is to be performed. Note that information indicating thespecimen thickness may be stored in the image server 1201 or estimatedfrom the AF information obtained in Step S1302. Further, informationindicating the staining method (or the tinge) may be stored in the imageserver 1201 or obtained by the image processing apparatus 102 during theimage processing. When a cytological diagnosis is determined, theroutine advances to Step S1304, and when a tissue diagnosis isdetermined, the routine advances to Step S1305.

In Step S1304, the image processing apparatus 102, having receivednotification of a cytological diagnosis, sets the Z interval of theselected images to be wide. The reason for this is that in a cytologicaldiagnosis, the thickness is several tens of μm. By widening the Zinterval (to between 1 and several μm, for example) in comparison withthat of a tissue diagnosis, the number of layer images used for focusstacking can be reduced, and as a result, the processing time can beshortened.

In Step S1305, the image processing apparatus 102, having receivednotification of a tissue diagnosis, sets the Z interval of the selectedimages to be narrow. The reason for this is that in a tissue diagnosis,the specimen thickness is only approximately several μm. When an NA ofthe imaging optical system of the imaging apparatus 101 is approximately0.7, the depth of field is approximately ±0.5 μm, and even then thespecimen thickness is larger. Hence, to obtain a sharp image, the Zinterval is preferably set at approximately 0.5 μm.

In Step S1306, the image processing apparatus 102 obtains the requiredlayer images from the image server 1201 in accordance with the Z rangedesignated in S1302 and the Z interval designated in Step S1304 orS1305. The observation image and the analysis image are then generatedusing similar processing to the first embodiment, whereupon requiredprocessing is performed on the respective images (S607 to S613).

As described above, according to this embodiment, similarly to the firstembodiment, two images, namely the observation image and the analysisimage, can be generated from identical Z-stack images (the layer imagegroup), enabling an improvement in user-friendliness. In this embodimentin particular, the observation image and the analysis image aregenerated on the basis of images obtained in advance, and therefore adesired image can be obtained without taking the imaging time requiredby the imaging apparatus into account. Further, the Z interval of theimages used for focus stacking can be adjusted automatically dependingon whether a tissue diagnosis or a cytological diagnosis is to beperformed, and therefore the processing time of a cytological diagnosiscan be shortened. Moreover, a further improvement in convenience can beprovided by determining whether a tissue diagnosis or a cytologicaldiagnosis is to be performed automatically.

Third Embodiment

A third embodiment of the present invention will now be described. Inthe above embodiments, focus stacking is performed using a select andmerge method, but in the third embodiment, focus stacking is implementedusing a spatial frequency filtering method in which original images areadded together in a spatial frequency region.

FIG. 11 is a flowchart showing a flow of focus stacking processing. FIG.11 shows in detail the content of Steps S805 and S810 in FIG. 6 of thefirst embodiment. In the first embodiment, divided images are selectedand merged by comparing contrast values. In this embodiment, processing(also referred to as depth recovery) for enlarging the depth of field byperforming image stacking and recovery in a frequency region using aplurality of images will be described.

In Step S1401, the image processing apparatus 102 obtains a plurality oflayer images to be subjected to depth recovery processing.

In Step S1402, the image processing apparatus 102 divides the obtainedimages respectively into a plurality of regions of a predetermined size.

In Step S1403, the image processing apparatus 102 selects images to beused for focus stacking in each divided region. Similarly to the firstand second embodiments, in the case of the observation image, aplurality of images are selected in a range where depth information ispreserved, while in the case of the analysis image, the number of imagesto be selected is determined in consideration of the thickness range inwhich overlap between the cells and nuclei serving as the analysissubjects is at a minimum. Image selection is based on a pre-designatedregion range and the pre-measurement results.

In Step S1404, the image processing apparatus 102 applies a Fouriertransform to the divided images selected in Step S1403. Although aFourier transform is cited here as an example of a spatial frequencytransform, other frequency transform processing such as a discretecosine transform may be used instead.

In Step S1405, the image processing apparatus 102 determines whether ornot the Fourier transform has been applied to all of the images. Whenthe Fourier transform has been applied to all of the images, the routineadvances to Step S1406. When an image to which the Fourier transform hasnot yet been applied exists, the routine returns to Step S1404, wherethe transform processing is applied to the next image.

In Step S1406, the image processing apparatus 102, having receivednotification that the frequency transform processing has been completedon all of the divided images, adds together frequency components ofdivided images in identical positions on the XY plane while applyingappropriate weightings thereto.

In Step S1407, the image processing apparatus 102 generates a focusedimage by subjecting a Fourier spectrum (image information) obtained fromthe weighted addition to an inverse Fourier transform. Here, an inversetransform from a frequency region to a spatial region is envisaged.

In Step S1408, the image processing apparatus 102 applies filterprocessing such as edge emphasis, smoothing, and noise removal asrequired. Note that this step may be omitted.

As described above, in this embodiment, similarly to the firstembodiment, two images, namely the observation image and the analysisimage, can be generated from identical Z-stack images (the layer imagegroup), enabling an improvement in user-friendliness. With the spatialfrequency filtering method of this embodiment in particular, a pluralityof images are added together in a frequency region rather thanexclusively selecting the image having the highest contrast value, as inthe select and merge method, and therefore a high-quality image can begenerated.

Other Embodiments

The object of the present invention may be achieved as follows. Arecording medium (or a storage medium) recorded with program code ofsoftware for realizing all or a part of the functions of the embodimentsdescribed above is supplied to a system or an apparatus. A computer (ora CPU or MPU) of the system or the apparatus then reads and executes theprogram code stored in the recording medium. In this case, the programcode read from the recording medium realizes the functions of the aboveembodiments, while the recording medium recorded with the program codeconstitutes the present invention.

Further, by having the computer execute the read program code, anoperating system (OS) or the like that runs on the computer performs allor a part of the actual processing on the basis of instructions includedin the program code. A case in which the functions of the aboveembodiments are realized by this processing is also included in thepresent invention.

Furthermore, the program code read from the recording medium is writtento a memory included in a function expansion card inserted into thecomputer or a function expansion unit connected to the computer. A CPUor the like included in the function expansion card or functionexpansion unit then performs all or a part of the actual processing onthe basis of instructions included in the program code, and a case inwhich the functions of the above embodiments are realized by thisprocessing is also included in the present invention.

When the present invention is applied to the recording medium describedabove, program code corresponding to the flowcharts described above isstored in the recording medium.

Further, in the first to third embodiments, an example in which twoimages, namely the observation image and the analysis image, aregenerated from a plurality of layer images having different focalpositions relative to the specimen was described, but the presentinvention is not limited thereto. For example, when the observationimage is not required (or when a layer image is used as is), theanalysis image may be generated alone. Furthermore, by controlling anaperture value of the imaging optical system of the imaging apparatusrather than enlarging the depth of field through focus stacking, animage having an enlarged depth of field can be obtained in accordancewith the application. Moreover, depth of field enlargement can beperformed on the analysis image by applying a typical depth recoverytechnique in which a single image and information indicating thedistance to the specimen are obtained instead of a plurality of layerimages and a PSF (point spread function) is estimated on the basis ofthe distance information.

Further, the constitutions described in the first to third embodimentsmay be combined with each other. For example, the focus stackingprocessing of the third embodiment may be applied to the systems of thefirst and second embodiments, and the image processing apparatus may beconnected to both the imaging apparatus and the image server so that animage to be used in the processing can be obtained from eitherapparatus. Other constitutions obtained by appropriately combining thevarious techniques described in the above embodiments also belong to thescope of the present invention.

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., non-transitory computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2011-155889, filed on Jul. 14, 2011, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: animage sensor, for acquiring a plurality of original images by imaging aspecimen including a structure while changing a focal position of amicroscope apparatus within an imaging range in an optical axisdirection; an image analyzer, for obtaining information relating to thestructure included in an image by applying image analysis processing tothe image; and an image generator, for generating, on the basis of theplurality of original images, a first image to be used for the imageanalysis processing by said image analyzer and a second image to be usedfor visual observation by a user, wherein said image generator sets up afirst depth range within the imaging range, the first depth range beingnarrower than the imaging range, selects only those original imageswhose focal positions are included within the first depth range out ofthe plurality of original images obtained from the specimen, andgenerates the first image using only the selected original images, andwherein said image generator sets up a second depth range within theimaging range, the second depth range being narrower than the firstdepth range, selects only those original images whose focal positionsare included within the second depth range out of the plurality oforiginal images obtained from the specimen, and generates the secondimage using only the selected original images.
 2. The image processingapparatus according to claim 1, wherein said image generator determinesa size of the first depth range to be approximately equal to a size ofthe structure to be analyzed in the image analysis processing.
 3. Theimage processing apparatus according to claim 2, wherein said imagegenerator determines the size of the first depth range, depending onwhether the specimen is for a tissue diagnosis or for a cytologicaldiagnosis, so that the depth range of the specimen for a tissuediagnosis is narrower than that of the specimen for a cytologicaldiagnosis.
 4. The image processing apparatus according to claim 2,wherein, when the specimen is to be used in a tissue diagnosis, saidimage generator sets the size of the first depth range, in which theoriginal images used to generate the first image are selected, at notless than 3 μm and not more than 10 μm.
 5. The image processingapparatus according to claim 1, wherein the depth range in which theoriginal images used to generate the first image are selected, isdesignated by a user.
 6. The image processing apparatus according toclaim 1, wherein said image generator generates the first image byperforming focus stacking processing using the selected original images.7. The image processing apparatus according to claim 6, wherein saidimage generator generates the first image by dividing the originalimages into a plurality of regions, selecting an image having highestcontrast value from among the selected original images in each dividedregion, and merging the images selected in each divided region with eachother.
 8. Image processing apparatus according to claim 6, wherein saidimage generator generates the first image by dividing the originalimages into a plurality of regions and adding the selected originalimages together in a spatial frequency region in each divided region. 9.The image processing apparatus according to claim 1, wherein thestructure is a cell.
 10. The image processing apparatus according toclaim 9, wherein information relating to the structure obtained by saidimage analyzer is at least any of a cell contour, a nucleus contour, acell count, a cell shape, a cytoplasm surface area, a nucleus surfacearea, and an area ratio between the cytoplasm and the nucleus.
 11. Theimage processing apparatus according to claim 1, wherein the structureis a nucleus.
 12. The image processing apparatus according to claim 1,wherein the second image is an image on which blur has been reduced to asmaller extent than the first image.
 13. The image processing apparatusaccording to claim 12, wherein the second image is an image to be usedfor visual observation by a user.
 14. The image processing apparatusaccording to claim 12, further comprising a unit for outputting theinformation relating to the structure obtained by said image analyzertogether with the second image to a display device.
 15. An imagingsystem comprising: a microscope apparatus for obtaining a plurality oforiginal images by imaging a specimen including a structure in variousfocal positions; and the image processing apparatus according to claim1, which obtains the plurality of original images from said microscopeapparatus.
 16. The imaging system according to claim 15, wherein saidmicroscope apparatus includes a measurement unit for measuring athickness of the specimen.
 17. The imaging system according to claim 16,wherein said microscope apparatus determines an imaging start position,from which the plurality of original images are obtained, on the basisof a measurement result from said measurement unit.
 18. The imagingsystem according to claim 15, wherein said microscope apparatus variesan imaging interval, at which the plurality of original images areobtained, as between a case in which the specimen is to be used in atissue diagnosis and a case in which the specimen is to be used in acytological diagnosis.
 19. An image processing system comprising: animage server for storing a plurality of original images obtained byimaging a specimen including a structure in various focal positions; andthe image processing apparatus according to claim 1, which obtains theplurality of original images from said image server.
 20. A computerprogram stored on a non-transitory computer-readable medium, the programcausing a computer to perform a method comprising the steps of:acquiring a plurality of original images acquired by imaging a specimenincluding a structure while changing a focal position of a microscopeapparatus within an imaging range in an optical axis direction;acquiring information relating to the structure included in an image byapplying image analysis processing to the image; and generating, on thebasis of the plurality of original images, a first image to be used forthe image analysis processing in said information acquiring step and asecond image to be used for visual observation by a user, wherein, inthe image generation step, a first depth range is set up within theimaging range, the first depth range being narrower than the imagingrange, only those original images whose focal positions are includedwithin the first depth range are selected out of the plurality oforiginal images obtained from the specimen, and the first image isgenerated using only the selected original images, and a second depthrange is set up within the imaging range, the second depth range beingnarrower than the first depth range, only those original images whosefocal positions are included within the second depth range are selectedout of the plurality of original images obtained from the specimen, andthe second image is generated using only the selected original images.